Intonation vs. Gestures in Yes-No Answers

Yes-No answers have traditionally posed major difficulties in speech recognition: the amount of signal is too short and may lead to mis-recognition. We think that the speech signal itself, that is, intonation (F0 and energy) are not enough to fulfil this task. When the information the signal can provide is so little, we must look for other sources. This paper analyzes intonation and gestures to process yes-no answers in real dialogues in a human-machine system. This study is especially interesting when nonlexical items are employed to substitute the words “yes” and “no” in casual spontaneous speech. We have studied 50 dialog extracts in Spanish. In our experiments trained evaluators report on the speakers’ answers. The evaluators relied mostly on gestures when the information was not clear.